{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Deterministic Policy Gradients (DDPG)\n",
    "\n",
    "In this tutorial we'll cover how to implement the Deep Deterministic Policy Gradients (DDPG), a policy-gradient actor-critic algorithm, that is off-policy and model-free. This work is based on [Patrick Emami's](https://github.com/pemami4911) previous write-up and code as described in the resources section.\n",
    "\n",
    "*This notebook might also be using [JSAnimation](http://nbviewer.jupyter.org/github/patrickmineault/xcorr-notebooks/blob/master/Render%20OpenAI%20gym%20as%20GIF.ipynb) which allows to include OpenAI gym environments as part of the notebooks*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Policy Gradient methods\n",
    "\n",
    "Policy-Gradient (PG) algorithms optimize a policy end-to-end by computing noisy estimates of the gradient of the expected reward of the policy and then updating the policy in the gradient direction. Traditionally, the so called *vanilla PG methods* assumed a stochastic policy of the type $\\mu (a|s)$, which gives a probability distribution over actions.  Policy gradient algorithms utilize a form of policy iteration: they evaluate the policy, and then follow the policy gradient to maximize performance. Ideally, the algorithm sees lots of training examples of high rewards from good actions and negative rewards from bad actions.\n",
    "\n",
    "For Reinforcement Learning (RL) problems with continuous action spaces, vanilla-PG is all but useless. You can, however, get vanilla-PG to work with some RL domains that take in visual inputs and have discrete action spaces with a convolutional neural network representing your policy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Deterministic Policy Gradients\n",
    "\n",
    "Deep Deterministic Policy Gradients (DDPG) is a policy gradient algorithm that uses a stochastic behavior policy for good exploration but estimates a **deterministic** target policy, which is much easier to learn. Since DDPG is off-policy and uses a deterministic target policy, this allows for the use of the Deterministic Policy Gradient theorem (which will be derived shortly). \n",
    "\n",
    "DDPG is an actor-critic algorithm, it primarily uses two neural networks, one for the actor and one for the critic. These networks compute action predictions for the current state and generate a temporal-difference (TD) error signal each time step. The input of the actor network is the current state, and the output is a single real value representing an action chosen from a **continuous** action space. The critic's output is simply the estimated Q-value of the current state and of the action given by the actor. The deterministic policy gradient theorem provides the update rule for the weights of the actor network. The critic network is updated from the gradients obtained from the TD error signal. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pendulum\n",
    "Pendulum environment in OpenAI gym; a low-dimensional state space and a single continuous action within [-2, 2] where the goal is to swing up and balance the pendulum."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check TF version and GPU support"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow Version: 1.1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/site-packages/ipykernel_launcher.py:14: UserWarning: No GPU found. Please use a GPU to train your neural network.\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "from distutils.version import LooseVersion\n",
    "import warnings\n",
    "import tensorflow as tf\n",
    "\n",
    "# reset everything to rerun in jupyter\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# Check TensorFlow Version\n",
    "assert LooseVersion(tf.__version__) >= LooseVersion('1.0'), 'Please use TensorFlow version 1.0 or newer'\n",
    "print('TensorFlow Version: {}'.format(tf.__version__))\n",
    "\n",
    "# Check for a GPU\n",
    "if not tf.test.gpu_device_name():\n",
    "    warnings.warn('No GPU found. Please use a GPU to train your neural network.')\n",
    "else:\n",
    "    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A replay buffer class\n",
    "The replay buffer class provides an abstraction to return a randomly chosen batch of experienced when queried. Refer to literature and the problem of experience replay."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "import random\n",
    "import numpy as np\n",
    "\n",
    "class ReplayBuffer(object):\n",
    "\n",
    "    def __init__(self, buffer_size, random_seed=123):\n",
    "        \"\"\"\n",
    "        The right side of the deque contains the most recent experiences\n",
    "        \"\"\"\n",
    "        self.buffer_size = buffer_size\n",
    "        self.count = 0\n",
    "        self.buffer = deque()\n",
    "        random.seed(random_seed)\n",
    "\n",
    "    def add(self, s, a, r, t, s2):\n",
    "        experience = (s, a, r, t, s2)\n",
    "        if self.count < self.buffer_size:\n",
    "            self.buffer.append(experience)\n",
    "            self.count += 1\n",
    "        else:\n",
    "            self.buffer.popleft()\n",
    "            self.buffer.append(experience)\n",
    "\n",
    "    def size(self):\n",
    "        return self.count\n",
    "\n",
    "    def sample_batch(self, batch_size):\n",
    "        batch = []\n",
    "\n",
    "        if self.count < batch_size:\n",
    "            batch = random.sample(self.buffer, self.count)\n",
    "        else:\n",
    "            batch = random.sample(self.buffer, batch_size)\n",
    "\n",
    "        s_batch = np.array([_[0] for _ in batch])\n",
    "        a_batch = np.array([_[1] for _ in batch])\n",
    "        r_batch = np.array([_[2] for _ in batch])\n",
    "        t_batch = np.array([_[3] for _ in batch])\n",
    "        s2_batch = np.array([_[4] for _ in batch])\n",
    "\n",
    "        return s_batch, a_batch, r_batch, t_batch, s2_batch\n",
    "\n",
    "    def clear(self):\n",
    "        self.deque.clear()\n",
    "        self.count = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Actor network - $\\mu(s | \\theta^\\mu)$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import gym\n",
    "from gym import wrappers\n",
    "import tflearn\n",
    "\n",
    "\n",
    "\n",
    "class ActorNetwork(object):\n",
    "    \"\"\"\n",
    "    Input to the network is the state, output is the action\n",
    "    under a deterministic policy.\n",
    "\n",
    "    The output layer activation is a tanh to keep the action\n",
    "    between -2 and 2\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, sess, state_dim, action_dim, action_bound, learning_rate, tau):\n",
    "        self.sess = sess\n",
    "        self.s_dim = state_dim\n",
    "        self.a_dim = action_dim\n",
    "        self.action_bound = action_bound\n",
    "        self.learning_rate = learning_rate\n",
    "        self.tau = tau\n",
    "\n",
    "        # Actor Network\n",
    "        with tf.name_scope('actor'):\n",
    "            self.inputs, self.out, self.scaled_out = self.create_actor_network()\n",
    "\n",
    "        self.network_params = tf.trainable_variables()\n",
    "\n",
    "        # Target Network\n",
    "        with tf.name_scope('target_actor'):\n",
    "            self.target_inputs, self.target_out, self.target_scaled_out = self.create_actor_network()\n",
    "\n",
    "        self.target_network_params = tf.trainable_variables()[\n",
    "            len(self.network_params):]\n",
    "\n",
    "        # Op for periodically updating target network with online network\n",
    "        # weights\n",
    "        self.update_target_network_params = \\\n",
    "            [self.target_network_params[i].assign(tf.multiply(self.network_params[i], self.tau) +\n",
    "                                                  tf.multiply(self.target_network_params[i], 1. - self.tau))\n",
    "                for i in range(len(self.target_network_params))]\n",
    "\n",
    "        # This gradient will be provided by the critic network\n",
    "        self.action_gradient = tf.placeholder(tf.float32, [None, self.a_dim])\n",
    "\n",
    "        # Combine the gradients here\n",
    "        self.actor_gradients = tf.gradients(\n",
    "            self.scaled_out, self.network_params, -self.action_gradient)\n",
    "\n",
    "        # Optimization Op\n",
    "        self.optimize = tf.train.AdamOptimizer(self.learning_rate).\\\n",
    "            apply_gradients(zip(self.actor_gradients, self.network_params))\n",
    "\n",
    "        self.num_trainable_vars = len(\n",
    "            self.network_params) + len(self.target_network_params)\n",
    "\n",
    "    def create_actor_network(self):\n",
    "        inputs = tflearn.input_data(shape=[None, self.s_dim])\n",
    "        net = tflearn.fully_connected(inputs, 400, activation='relu')\n",
    "        net = tflearn.fully_connected(net, 300, activation='relu')\n",
    "        # Final layer weights are init to Uniform[-3e-3, 3e-3]\n",
    "        w_init = tflearn.initializations.uniform(minval=-0.003, maxval=0.003)\n",
    "        out = tflearn.fully_connected(\n",
    "            net, self.a_dim, activation='tanh', weights_init=w_init)\n",
    "        # Scale output to -action_bound to action_bound\n",
    "        scaled_out = tf.multiply(out, self.action_bound)\n",
    "        return inputs, out, scaled_out\n",
    "\n",
    "    def train(self, inputs, a_gradient):\n",
    "        with tf.name_scope('train_actor'):\n",
    "            self.sess.run(self.optimize, feed_dict={\n",
    "                self.inputs: inputs,\n",
    "                self.action_gradient: a_gradient\n",
    "            })\n",
    "\n",
    "    def predict(self, inputs):\n",
    "        return self.sess.run(self.scaled_out, feed_dict={\n",
    "            self.inputs: inputs\n",
    "        })\n",
    "\n",
    "    def predict_target(self, inputs):\n",
    "        return self.sess.run(self.target_scaled_out, feed_dict={\n",
    "            self.target_inputs: inputs\n",
    "        })\n",
    "\n",
    "    def update_target_network(self):\n",
    "        self.sess.run(self.update_target_network_params)\n",
    "\n",
    "    def get_num_trainable_vars(self):\n",
    "        return self.num_trainable_vars\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Critic network - $Q(s,a| \\theta^Q)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class CriticNetwork(object):\n",
    "    \"\"\"\n",
    "    Input to the network is the state and action, output is Q(s,a).\n",
    "    The action must be obtained from the output of the Actor network.\n",
    "\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, sess, state_dim, action_dim, learning_rate, tau, num_actor_vars):\n",
    "        self.sess = sess\n",
    "        self.s_dim = state_dim\n",
    "        self.a_dim = action_dim\n",
    "        self.learning_rate = learning_rate\n",
    "        self.tau = tau\n",
    "\n",
    "        # Create the critic network\n",
    "        with tf.name_scope('critic'):\n",
    "            self.inputs, self.action, self.out = self.create_critic_network()\n",
    "\n",
    "        self.network_params = tf.trainable_variables()[num_actor_vars:]\n",
    "\n",
    "        # Target Network\n",
    "        with tf.name_scope('target_critic'):\n",
    "            self.target_inputs, self.target_action, self.target_out = self.create_critic_network()\n",
    "\n",
    "        self.target_network_params = tf.trainable_variables()[(len(self.network_params) + num_actor_vars):]\n",
    "\n",
    "        # Op for periodically updating target network with online network\n",
    "        # weights with regularization\n",
    "        self.update_target_network_params = \\\n",
    "            [self.target_network_params[i].assign(tf.multiply(self.network_params[i], self.tau) + tf.multiply(self.target_network_params[i], 1. - self.tau))\n",
    "                for i in range(len(self.target_network_params))]\n",
    "\n",
    "        # Network target (y_i)\n",
    "        with tf.name_scope('y_i'):\n",
    "            self.predicted_q_value = tf.placeholder(tf.float32, [None, 1])\n",
    "\n",
    "        # Define loss and optimization Op        \n",
    "        with tf.name_scope('loss'):\n",
    "            self.loss = tflearn.mean_square(self.predicted_q_value, self.out)\n",
    "        \n",
    "        with tf.name_scope('train'):\n",
    "            self.optimize = tf.train.AdamOptimizer(\n",
    "                self.learning_rate).minimize(self.loss)\n",
    "\n",
    "        # Get the gradient of the net w.r.t. the action.\n",
    "        # For each action in the minibatch (i.e., for each x in xs),\n",
    "        # this will sum up the gradients of each critic output in the minibatch\n",
    "        # w.r.t. that action. Each output is independent of all\n",
    "        # actions except for one.\n",
    "        self.action_grads = tf.gradients(self.out, self.action)\n",
    "\n",
    "    def create_critic_network(self):\n",
    "        with tf.name_scope('inputs'):\n",
    "            inputs = tflearn.input_data(shape=[None, self.s_dim])\n",
    "        with tf.name_scope('action'):    \n",
    "            action = tflearn.input_data(shape=[None, self.a_dim])            \n",
    "        net = tflearn.fully_connected(inputs, 400, activation='relu')\n",
    "\n",
    "        # Add the action tensor in the 2nd hidden layer\n",
    "        # Use two temp layers to get the corresponding weights and biases\n",
    "        t1 = tflearn.fully_connected(net, 300)\n",
    "        t2 = tflearn.fully_connected(action, 300)\n",
    "\n",
    "        net = tflearn.activation(\n",
    "            tf.matmul(net, t1.W) + tf.matmul(action, t2.W) + t2.b, activation='relu')\n",
    "\n",
    "        # linear layer connected to 1 output representing Q(s,a)\n",
    "        # Weights are init to Uniform[-3e-3, 3e-3]\n",
    "        w_init = tflearn.initializations.uniform(minval=-0.003, maxval=0.003)\n",
    "        out = tflearn.fully_connected(net, 1, weights_init=w_init)\n",
    "        return inputs, action, out\n",
    "\n",
    "    def train(self, inputs, action, predicted_q_value):\n",
    "        with tf.name_scope('train_critic'):\n",
    "            return self.sess.run([self.out, self.optimize], feed_dict={\n",
    "                self.inputs: inputs,\n",
    "                self.action: action,\n",
    "                self.predicted_q_value: predicted_q_value\n",
    "            })\n",
    "\n",
    "    def predict(self, inputs, action):\n",
    "        return self.sess.run(self.out, feed_dict={\n",
    "            self.inputs: inputs,\n",
    "            self.action: action\n",
    "        })\n",
    "\n",
    "    def predict_target(self, inputs, action):\n",
    "        return self.sess.run(self.target_out, feed_dict={\n",
    "            self.target_inputs: inputs,\n",
    "            self.target_action: action\n",
    "        })\n",
    "\n",
    "    def action_gradients(self, inputs, actions):\n",
    "        return self.sess.run(self.action_grads, feed_dict={\n",
    "            self.inputs: inputs,\n",
    "            self.action: actions\n",
    "        })\n",
    "\n",
    "    def update_target_network(self):\n",
    "        self.sess.run(self.update_target_network_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Auxiliary functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train(sess, env, actor, critic):\n",
    "\n",
    "    # Set up summary Ops\n",
    "    summary_ops, summary_vars = build_summaries()\n",
    "\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    writer = tf.summary.FileWriter(SUMMARY_DIR, sess.graph)\n",
    "\n",
    "    # Initialize target network weights\n",
    "    actor.update_target_network()\n",
    "    critic.update_target_network()\n",
    "\n",
    "    # Initialize replay memory\n",
    "    replay_buffer = ReplayBuffer(BUFFER_SIZE, RANDOM_SEED)\n",
    "\n",
    "    for i in range(MAX_EPISODES):\n",
    "\n",
    "        s = env.reset()\n",
    "\n",
    "        ep_reward = 0\n",
    "        ep_ave_max_q = 0\n",
    "\n",
    "        for j in range(MAX_EP_STEPS):\n",
    "\n",
    "            if RENDER_ENV:\n",
    "                env.render()\n",
    "\n",
    "            # Added exploration noise\n",
    "            a = actor.predict(np.reshape(s, (1, 3))) + (1. / (1. + i))\n",
    "\n",
    "            s2, r, terminal, info = env.step(a[0])\n",
    "\n",
    "            replay_buffer.add(np.reshape(s, (actor.s_dim,)), np.reshape(a, (actor.a_dim,)), r,\n",
    "                              terminal, np.reshape(s2, (actor.s_dim,)))\n",
    "\n",
    "            # Keep adding experience to the memory until\n",
    "            # there are at least minibatch size samples\n",
    "            if replay_buffer.size() > MINIBATCH_SIZE:\n",
    "                s_batch, a_batch, r_batch, t_batch, s2_batch = \\\n",
    "                    replay_buffer.sample_batch(MINIBATCH_SIZE)\n",
    "\n",
    "                # Calculate targets\n",
    "                target_q = critic.predict_target(\n",
    "                    s2_batch, actor.predict_target(s2_batch))\n",
    "\n",
    "                y_i = []\n",
    "                for k in range(MINIBATCH_SIZE):\n",
    "                    if t_batch[k]:\n",
    "                        y_i.append(r_batch[k])\n",
    "                    else:\n",
    "                        y_i.append(r_batch[k] + GAMMA * target_q[k])\n",
    "\n",
    "                # Update the critic given the targets\n",
    "                predicted_q_value, _ = critic.train(\n",
    "                    s_batch, a_batch, np.reshape(y_i, (MINIBATCH_SIZE, 1)))\n",
    "\n",
    "                ep_ave_max_q += np.amax(predicted_q_value)\n",
    "\n",
    "                # Update the actor policy using the sampled gradient\n",
    "                a_outs = actor.predict(s_batch)\n",
    "                grads = critic.action_gradients(s_batch, a_outs)\n",
    "                actor.train(s_batch, grads[0])\n",
    "\n",
    "                # Update target networks\n",
    "                actor.update_target_network()\n",
    "                critic.update_target_network()\n",
    "\n",
    "            s = s2\n",
    "            ep_reward += r\n",
    "            tf.summary.scalar('ep_reward', ep_reward)\n",
    "            \n",
    "            if terminal:\n",
    "\n",
    "                summary_str = sess.run(summary_ops, feed_dict={\n",
    "                    summary_vars[0]: ep_reward,\n",
    "                    summary_vars[1]: ep_ave_max_q / float(j)\n",
    "                })\n",
    "\n",
    "                writer.add_summary(summary_str, i)\n",
    "                writer.flush()\n",
    "\n",
    "                print('| Reward: %.2i' % int(ep_reward), \" | Episode\", i, \\\n",
    "                    '| Qmax: %.4f' % (ep_ave_max_q / float(j)))\n",
    "\n",
    "                break\n",
    "\n",
    "                \n",
    "def build_summaries():\n",
    "    episode_reward = tf.Variable(0.)\n",
    "    tf.summary.scalar(\"Reward\", episode_reward)\n",
    "    episode_ave_max_q = tf.Variable(0.)\n",
    "    tf.summary.scalar(\"Qmax_value\", episode_ave_max_q)\n",
    "\n",
    "    summary_vars = [episode_reward, episode_ave_max_q]\n",
    "    summary_ops = tf.summary.merge_all()\n",
    "\n",
    "    return summary_ops, summary_vars"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-05-17 11:27:04,034] Making new env: Pendulum-v0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Error encountered when serializing data_preprocessing.\n",
      "Type is unsupported, or the types of the items don't match field type in CollectionDef.\n",
      "'NoneType' object has no attribute 'name'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-05-17 11:27:07,968] Error encountered when serializing data_preprocessing.\n",
      "Type is unsupported, or the types of the items don't match field type in CollectionDef.\n",
      "'NoneType' object has no attribute 'name'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Error encountered when serializing data_augmentation.\n",
      "Type is unsupported, or the types of the items don't match field type in CollectionDef.\n",
      "'NoneType' object has no attribute 'name'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-05-17 11:27:07,970] Error encountered when serializing data_augmentation.\n",
      "Type is unsupported, or the types of the items don't match field type in CollectionDef.\n",
      "'NoneType' object has no attribute 'name'\n",
      "[2017-05-17 11:27:08,997] Starting new video recorder writing to /Users/Victor/basic_reinforcement_learning/tutorial9/results/gym_ddpg/openaigym.video.1.24646.video000000.mp4\n",
      "[2017-05-17 11:27:25,014] Starting new video recorder writing to /Users/Victor/basic_reinforcement_learning/tutorial9/results/gym_ddpg/openaigym.video.1.24646.video000001.mp4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| Reward: -1062  | Episode 0 | Qmax: -0.2586\n",
      "| Reward: -1686  | Episode 1 | Qmax: -0.5576\n",
      "| Reward: -952  | Episode 2 | Qmax: -0.6692\n"
     ]
    }
   ],
   "source": [
    "# ==========================\n",
    "#   Training Parameters\n",
    "# ==========================\n",
    "# Max training steps\n",
    "MAX_EPISODES = 50000\n",
    "# Max episode length\n",
    "MAX_EP_STEPS = 1000\n",
    "# Base learning rate for the Actor network\n",
    "ACTOR_LEARNING_RATE = 0.0001\n",
    "# Base learning rate for the Critic Network\n",
    "CRITIC_LEARNING_RATE = 0.001\n",
    "# Discount factor\n",
    "GAMMA = 0.99\n",
    "# Soft target update param\n",
    "TAU = 0.001\n",
    "\n",
    "# ===========================\n",
    "#   Utility Parameters\n",
    "# ===========================\n",
    "# Render gym env during training\n",
    "RENDER_ENV = True\n",
    "# Use Gym Monitor\n",
    "GYM_MONITOR_EN = True\n",
    "# Gym environment\n",
    "ENV_NAME = 'Pendulum-v0'\n",
    "# Directory for storing gym results\n",
    "MONITOR_DIR = './results/gym_ddpg'\n",
    "# Directory for storing tensorboard summary results\n",
    "SUMMARY_DIR = './results/tf_ddpg'\n",
    "RANDOM_SEED = 1234\n",
    "# Size of replay buffer\n",
    "BUFFER_SIZE = 10000\n",
    "MINIBATCH_SIZE = 64\n",
    "\n",
    "with tf.Session() as sess:\n",
    "\n",
    "    env = gym.make(ENV_NAME)\n",
    "    np.random.seed(RANDOM_SEED)\n",
    "    tf.set_random_seed(RANDOM_SEED)\n",
    "    env.seed(RANDOM_SEED)\n",
    "\n",
    "    state_dim = env.observation_space.shape[0]\n",
    "    action_dim = env.action_space.shape[0]\n",
    "    action_bound = env.action_space.high\n",
    "    # Ensure action bound is symmetric\n",
    "    assert (env.action_space.high == -env.action_space.low)\n",
    "\n",
    "    with tf.name_scope('actor_network'):\n",
    "        actor = ActorNetwork(sess, state_dim, action_dim, action_bound,\n",
    "                         ACTOR_LEARNING_RATE, TAU)\n",
    "\n",
    "    with tf.name_scope('critic_network'):        \n",
    "        critic = CriticNetwork(sess, state_dim, action_dim,\n",
    "                           CRITIC_LEARNING_RATE, TAU, actor.get_num_trainable_vars())\n",
    "\n",
    "    if GYM_MONITOR_EN:\n",
    "        if not RENDER_ENV:\n",
    "            env = wrappers.Monitor(\n",
    "                env, MONITOR_DIR, video_callable=False, force=True)\n",
    "        else:\n",
    "            env = wrappers.Monitor(env, MONITOR_DIR, force=True)\n",
    "\n",
    "    train(sess, env, actor, critic)\n",
    "\n",
    "    if GYM_MONITOR_EN:\n",
    "        env.monitor.close()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensorboard, displaying the network\n",
    "Displaying the network with tensorboard outputs something as follows:\n",
    "\n",
    "![](././tensorboard-vis2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resources:\n",
    "- http://pemami4911.github.io/blog/2016/08/21/ddpg-rl.html#Tensorflow\n",
    "- [Deep Deterministic Policy Gradients (DDPG) paper](https://arxiv.org/pdf/1509.02971v2.pdf)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
